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SYSTEM AND METHOD FOR INDEXING RECORDINGS OF OBSERVED T EST 

EVENTS AND ASSESSED PHENOMENA USING PRE DEFINED MEASUREMENT 

ITEMS 

This application is a continuation in part of provisional patent application serial no. 
60/127,194 on March 31, 1999, the disclosure of which is herein incorporated by reference. 

This invention was supported m adg with a grant from the U.S. government which may 
have rights to the invention. 

COPYRIGHT NOTIFICATI ON 

Portions of this patent application contain mat e rials that are subj e ct to copyright 
prot e ction. Th e copyright own e r has no objection to th e facsimil e reproduction by anyon e of 
th e pat e nt document or patent disclosur e , as it appears in the Patent and Trademark Offic e 
patent file or records, but oth e rwise r es erv e s all copyright right s- whatsoever. 

FIELD OF THE INVENTION 

Th e pr e s e nt inv e ntion relates to information proc e ssing syst e ms and obs e rvation and 
a s s e ssm e nt m e thods and op e rations, including th e training and monitoring of int e rvi e woro or 
obs e rv e r s , quantitativ e and qualitativ e rating and ass e ssment proc e ss e s, and the storage and 
retrieval of recordings of int e rvi e ws or obs e rv e d e v e nts and, mor e particularly, to a syst e m 
and method for ind e xing recordings of interviews or other obs e rv e d e v e nts or ph e nomena by 
combining systematic us e by an observer of pro defined descriptive or quantitative 
m e asur e m e nt items with simultaneous digital recording to automatically g e n e rat e ind e x e s. 

The present invention generally relates to information processing systems. More 
particularly, the invention is a data capture and a ccess system that correla tes data with 
recorded signals with an index based on pre-iden tified identifiers at the time of capture but 
allows later annotation or additional indexing without affecting the original index identifiers 
or the recorded signals. The invention specifically provides a method for indexing of digital 
recordings of events bv use of ore-entered or ore-e xisting measu rement items that serve as 
index information designators such as kev words or labels. Such a method is particularly 
useful as those used in a systematic interview or assessment method. 
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DESCRIPTION OF THE PR IOR RELATED ART 

j aManv events in the medical, legalrs oientifio , business and education domains, 
among others, it is n e c e ssary - to systematically observe and describe, ass e ss, rat e , or oth e rwise 
measur e e v e nts or proc e ss e s (i. e ., ph e nom e na). It i s- also u s eful - to make recordings of th e s e 
e v e nts and to b e abl e to playback th e s e recordings. - For example, physicians may us e audio ? 
vid e o, or e lectrophysiological r e cording of an interview, a physical examination, a diagnostic 
procedur e , an operation, or autoDsv. t odav are bein g di gitally recorded. Physicians mav record 
a physical examination, operation, or autopsy and-may need to r e trieve and e xamine specific 
part s of a nnotate that recording at some later time for various purpos e s. Attorn e ys may carry 
ettfcga dto notes, observations, or other content in order to make a more complete record or 
draw previously unsee n trends. Lawyers mav seek to record a witness deposition and later 
playback s e ctions ofl infc that oral testimony in ord e r to link th e testimon y to - oth e rt g evidence 
(documentary, video, or testimony by other witnesses) in order to draw parallels, or highlight 
inconsistencie s, or for oth e r purpos e s. Social scientists apply systematic observation and 
rating m e thods - to - study soci e ty and culture . The frequency of such recordings, audio, video, 
or other, is likely to increase as their usefulness continues to gro w. In g e neral, th e u se fuln e ss 
depends upon th e purposes and value of retri e val and playback, th e rang e of r e tri e val arid 
playback functions, and how accurately and effici e ntly retrieval and playback can bo carriod 
out. Purposes includ e : thereby allowing for review of the assessor's conclusion or diagnoses. 
providing an educational or teaching tool to illustrate obs e rvation, assessment, and 
m e asurement techniques in general, cr e ating teaching mat e rials; monitoring teachers, 
trainees, and rat e rs; allowing for review of the assessor's conclusion or diagnoses; supporting 
r e tri e val of multipl e interviews, events, or segments of interviews or ev e nts for comparison 
and other studv: reviewing anoptimum assessment interview tec hniques, doc umenting the 
interview for legal and/or other purpose s; and other similar and varied uses in many domains . 

As a mor e d e tail e d and prototypical e xampl e , m e dicine M edicine and public health 
are -domains in which considerable effort has been devoted to the development of tools (o.g.± 
qu e stionnair e s or a ssessment schedules and i nstruments* for conducting syst e matic 
assessment interviews and e xaminations. A mai n . The objective of these efforts has often 
been to improve the accuracy, re liability^ and efficiency of a clinician's diagnosis of ahis/her 
patient 

Among medical disciplines, psychiatry researchers have perhaps allocated the most 
efforts to developing patient assessment m e thods and tools g jds . Examples of the types of 
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ass e ssm e nt instruments that hav e be e n d e v e lop e d, which ar e compos e d of d e scriptor or 
quantitativ e assessment (i. e ., measurement) item s or questions. instruments developed include 
the Structured Clinical Interview for DSM (SCID), the Diagnostic Interview Schedule (DIS), 
the Composite International Diagnostic Interview (CIDI), the World Health Organization 
(WHO) International Personality Disorders Examination (IPDE), and the WHO Schedules for 
Clinical Assessment in Neuropsychiatry (SCAN). These tests have been standardized around 
s e ts of inquiri e s or m e asurem e nt it e ms. Measurement items typically ar e composed ofi 1) 
definition of a phenom e non of int e r e st; 2) instructions r e garding obs e rvation of th e 
phenomenon; and 3) a quantitativ e rating scal e . Th e scientific fi e ld of measur e m e nt, known 
a s- psyohometrics , includ e s study of and m e thods p e rtaining to syst e matic and standardized 
o bservation^ measurement, training, monitoring, - and comparison. Psychom e tric issu e s of 
importanc e include efficiency, reliability (consistency), and validity ( accuracy). a core set of 
inquiries or directions. 

The response to or rating of standardized question s, inquiri e s, or measur e ment item s 
and instruction s or directions is of interes t for the assessor, both in terms of the ass e ssm e nt 
result ssubstantive response as well as the behavior manifested in the delivery and o f the 
assessment and in the r esponse. A s- w e l l- as e xpress e d symptoms. psvchiatrist sP svchiatrists 
may want to measure- or-rate th e behavioral response s of pati e nts, or may - wish to observe-aftd 
fate the actions of ethe^-psychiatrists in training. Cardiologists may want to measure physical 
responses to changes in applied stress, such as ohang e s in the EKG readings during a cardiac 
stress test. Lawyers may seek to identify inconsistencies between witnesses to an accident 
using standardized questions. Similar examples are found in virtually every profession where 
a pattern of syst e matio observation and t esting for response? and analysis of thatthe response? 
is us e d. In all th e se fi e lds, analog or digital recordings of audio, vid e o, physiologic, or oth e r 
asp e cts of obs e rv e d e v e nts ar e used to assist in these processe s is used . 

However, most-anyone who has participated in r e trieving and organizin gr eviewing 
these interview or testing recorded mat e rials for r e vi e w and stud v records realizes that the 
retri e val - and r e vi e w proc e ss is usually tedious, oft e n incomplet e and inaccurate, and ha s vast 
room for improv e m e nt. What is n ee d e d for improved retrieval and management of audio, 
video, or other recorded mat e rials ar e ind e x e s that ar e meaningful, accurate, and easily 
gen e rat e d, manipulat e d, and e xtended. Ind ee d, on e of the mo s t important features of any 
useful r e cording and r e tri e val review process needs help and improvement. If there was .an 
efficient wav to index the video, audio or other type of recording when the interview was 
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recorded, much time c ould be saved when later searching the records for particular 
information. 

It is. therefore, an object of the present invention to provide a system and method for 
conductin g recording, plaving back and editing or annotating digital recordings in audio, 
video, or other format. 

Kev to anv u seful recording system based on digital media is the ability to efficiently 
generate meaningful indexes for the recorded material. Such indexes would b e e sp e cially 
valuabl e for structur e d or systematic information gath e ring proc e ss e s in which professional or 
sci e ntific standards or e r e quired and/or in which considerabl e e fforts are mad e in training, 
quality control, and oth e r administrative functions, e.g., a t e l e vision intervi e w based upon a 
list of qu e stions, giving a natien ta re well suited for structured information gathering events 
that use a systematic ouestion-and-answer method 

containing a set or list of standardized inquiries, e.g.. a list of television interview Questions, a 
health questionnaire, a clinical or research medical assessment, conducting a set of 
deoositiorfc-us mg questions, a list of actions to activate brain function in different ways, 
conducting focus groups, e thnographic res e arch, or administ e rin g and a graded cardiac stress 
test, among others. 

In the SeWsfidd of medicine and health, the established means of recording 
structurecL assessments has generally been by use of analog a udio or video tape recorders. 
Indexing, if used at all, has been g e nerally t hrough means s uch as length counters and 
manually added entrie s, or with hybrid combinations of tape recorders with microprocessor 
sy s tems. This mak e s . This made the process of cr e ating indexes tedious and often incomplete. 
In addition, e ven aft e r ind e x e s ar e cr e at e d, t he use of tape recorders means that the process of 
searching and playback r emainsis slow and inefficien t. Finally, cr e ation of g e n e rally 
meaningful indexes has b e en probl e matic . 

In die field of digital recording, many indexing schemes have been developed, 
ranging from low-level encoding to linking of text with video frames. Industry standard 
digital encoding methods such as the MPEG-2 (Society of Motion Picture Engineers) format, 
include channels for indexing based upon time code. However, most of these approaches 
generate index entries in a form not directly useful for r e tri e val, p layback? and management, 
but rather provide basic mechanisms for supporting indexing . For e xampl e , on e prior art 

One system is known that uses computer voice-to-text processing to automatically 
generate words that are then used to index the s ource digital audio recording. Although ef 
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som e us eyggful, it does not provide a more organized level of indexing, since many words 
will be used in many areas of a recording, and hence as such d o not provide specific indexing 
to unique areas or segments of a recording. For indexing to be more useful, the ind e x e s or 
index items should contain meaningful and unique information that is associated with 
mrieuethe recording locations or s e gm e nts. -Maft vlocation or segment. Anv indexing 
methodsmethod can support this, but it is g e n e rally t edious and inefficient to generate such 
index items manually. j 

Exampl e s of prior art ind e xing syst e ms includ e U.S. Pat. No 1, 8 4 1,3 8 7, which 
disclos e s an indexing system using a touchpad to corr e lat e simultan e ously r e cord e d audio or 
vid e o information to handwritten - not e s mad e on numb e red sh ee ts of pap e r manually plac e d 
on top of th e touchpad. This system is simply a means for r e pla y- of - record e d audio or video 
information through manual e ntry of sheet - numbers and physical contact with th e pap e r in th e 
approximat e ar e a of th e original handwritten note, to generate a touchpad s ignal that is us e d 
to access the correlation ind e xed recording. Th e system do e s not anticipat e corr e lation of 
recordings with any information oth e r than th e position of handwritt e n notes and a pag e 
numb e r, nor does it - provid e any other means of - id e ntifying, s e arching, or r e tri e ving r e corded 
materials, nor do e s it support psychometric and mor e g e n e ral obs e rvation, measur e m e nt, 
retri e val, - training, monitoring, or comparison functions and operations. In th e professional 
and technical ar e nas, it is syst e matic as w e ll as quick acc e ss to sp e cific passag e s or segments 
of - reoorded material that d e t e rmin e s a syst e m's us e fulness, which is not ad e quately provided 
by utilizing the position of - hondwritton notes for ind e xing, y 

European patent application publication EP 0 495 612 disclos e s an ind e xing system 
that conn e cts manually e nter e d comput e r notes to - audio or video recordings, and p e rmits th e 
user to retriev e and play sections of a recording by s e lecting a note. The ind e xing is bas e d 
upon spatial location and organization of a s e t of notes within a document rather than th e 
m e aning of th e not e s, r e quiring th e us e r to manually associat e notes by physical location 
within the electronic docum e nt. B e caus e of this limitation, wh e n a user wish e s to assooiate - a 
segment of the recording with an e arli e r note, th e user must navigate to th e pag e and location 
in the existing set of notes, pr e s e nting a serious in e fficiency with lengthy note s . This system 
is also limited with regard to real - time augmentation of pr e viously e ntere d- notes during 
r e cording, since the later notes will b e tim e stamp e d with th e time they w e re entered rather 
than with - tho time of th e earli e r mat e rial to which th e y r e lat e or ar e r e l e vant, lacking any 
automatic index linkag e to th e e arli e r mat e rial. As with U.S. Pat. No. 1,81 1,3 8 7, this syst e m 
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r e quir e s manual entry of notes, do e s not standardiz e the proces s of note - taking nor provid e 
any s yst e matic means of obs e rvation, description, or assessm e nt, do e s not combin e 
s yst e matic assessment with gen e ration of ind e x e s, do e s not provid e a m e ans for quantitative 
rating of obs e rved event s , and do e s not provid e a m e ans of r e tri e val and playback of r e cord e d 
materials combined with syst e matic ro assessment or monitoring. 

U.S. Pat No. 5,172,2 8 1 disolosoo a system that supports an op e rator in manually 
corr e lating previously r e cord e d vid e o material with a standard t e xt transcript for th e purpose 
of g e nerating an ind e x that can th e n be used by s oftware that searches t e xt and finds 
ossociated vid e o ind e x e s. This syst e m has no capacity to operat e in real - tim e situation s . It 
do e s not provid e means for automated d e finition of segm e nt s of record e d material, nor do e s it 
provid e an automat e d m e ans of g e nerating th e actual ind e xes. It search e s th e text mat e rial 
only based - upon - op e rator e nter e d k e y word comments, and is unabl e to tak e advantag e of all 
th e information in th e original t e xt. It is subj e ct to th e probl e m of sp e cific comm e nt words 
b e ing us e d for more than one indexed timecode, requiring the operator who generate s the 
indexes to keep track of and ex e rcis e judg e ment - about - th e choic e of comm e nt words^ a 
significant w e akn e ss b e caus e it does not provid e a moans of handling multiple occurrences of 
ind e x k e y words in th e transcript material. 

U.S. Pat No. 4,03^387 di s close s a syst e m by which a st e nograph e r can control a 
vid e o cam e ra so as to st e nograph ieally record a deposition whil e simultan e ously op e rating a 
video camera and record e r. This system's main - objective i s to provid e a sy s tem to accurat e ly 
r e cord a deposition, not to e ffici e ntly and precisely looate - and pres e nt s pecific pas s ages of th e 
vid e otap e d e position. It provid e s a limit e d ind e xing structur e sinc e th e strok e r e cord mad e- is 
int e nd e d to b e a v e rbatim transcription of th e v e rbal t e stimony mod e in a courtroom, rather 
than any sy s tematic observation s or ass e ssm e nt of th e proc ee dings in th e courtroom. 

For all thes e syst e ms d e scrib e d abov e , th e user's ability to ind e x not e s or comm e nts 
to th e recording i s entirely controll e d through th e indicia, or not e s, th e us e r has manually 
e nt e r e d in a document. For both real - time and post proce s sing, these systems require the user 
first to proc e ss m e ntally the recorded event s and then generate and enter text to create notes, 
subj e cting the system and its - indexe s to individual user variance in generating the t e xt 
content. No method is suggested for facilitating or enhancin g a user' s personal - not e- taking 
styl e or accommodating the note - taking function s towards mor e s tandardiz e d - and systematic 
observation and assessment applications. In all th e s e syst e ms, no m e thod is e xplicitly 
suggested for handling the cas e of rep e tition or e xt e nsion of certain events or activities in a 
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oinglo rooording-or across multiple recordings, using a common indica for relat e d e vents or 
ph e nom e na. 

Existing systems supp ort some forms of inde xing of recorded signals in a variety of 
wavs. European p atent applica tion publication EP 0 495 612 bv La mming discloses a 
computer-based note-taking s ystem integrated with a n audio or video recording system. The 
computer presents a document editor stvle user interface to the user who either creates a new 
document or retr ieves an exis ting document to whic h the user adds notes as a recording is 
made or plaved via the integrated audio or video system. As the user enters each note (mark 
or indicumV the indicum is added to the document and it is time stamped and stored in an 
information-to-time-stamp i ndex. The time stamps are not visible to the user, thev are stor^4 
with the computer's internal representation of the indica entered bv the user. A video-frame 
time stamp function time stamps time code data received from the audio or video recorder 
and creates a time $tamp-to-time-code index. A browser function permits the user to retrieve 
sections of the recording using the indicia directly bv selecting the indicia. The browser looks 
up die indicia in the first index to retrieve the time stamp, and looks up the time code of the 
recording i n the second index using the time stamp, playing the section of the recording in the 
area indicated bv the time code. EP 0495 612 also discloses how time stamping the indicia 
mav be applied to creating topic or kev word data. Bv entering new, separate indicia spatially 
near a previously enter ed indicia tha t is a kev word or tonic whenever an idea or sneaker or 
topic applies to the previously ente red indicia, later selection of all of the marks spatially 
associated with a topic will result in all sectio ns of the recording indexed bv the time stamps 
of the respective indicia to be replayed. 

U.S. Pat No 4.841.387. entitled "Arrangement for Recording and Indexing 
Information" and issued to Rindfbss, discloses a system for recording information relating to 
an event on a recording medium, such as an audio or video tape, and for indexing positions of 
handwritten notations made on a touch sensitive device and concerning the event to positions 
on the recording medium in order to allow the user to identify portions of the handwritten 
notations for which review of the correlated material on the recorded medium is desired. In 
the recording mode, the device makes an audio recording of the event on a standard cassette 
tape. Simultaneously, the electronic touchoad senses the position of the user's handwritten 
notes on the writing surface, and provides this information to the microprocessor which 
correlates the record of the positions of the handwritten notations on each page with the 
position of the recorded information on the audio tape at corresponding j pftants in time. Real- 
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time constrained correlation vectors representing rectangular areas of the display surface each 
containing a cohesive group of handwritten text are each combined with a tape position that 
correlates to the instant i n time the handwriting within that area beg an. 

U.S. Pat. No. 4.425.586 iss ued to Miller discloses a system that combines a video tape 
recorder with a computer in such a manner that these two components each automatically 
record and display where related information is stored in its own mechanism as well as in its 
counterpart mechanism, allowing the user to determine the location of all the corresponding 
data stored both on video tape and on a storage medium such as a diskette bv examining only 
one storage medium. Notes about the recorded event or document mav be entered onto the 
diskette along with the automatic e ntry of the corresponding reel number and frame number 
of the video record and diskette and file address number of the computer storage medium. 
Another feature disclosed is the capability of t he system to enter and display the time and 
date on both the video tape and diskette recording mediums as well as on both video monitors 
along with the data address location information. 

U.S. Pat No. 4.924.387 issued to Jeppeson discloses a computerized court reporting; 
system which provides for periodically annotating the stroke record made bv the user of q 
court stenographic machine with a time stamp from a system clock while simultaneously 
sending a time stamp to a recording system making a vid eo and audio recording , of the 
testimony. The logic of a control system determines autom atically when to time stamn the 
stroke record and per mits the user to trigger a control function to annotate the video recording 
with autom atic "on th e record" and "off the record" messages with associated time stamps. 

U.S. Pat. No. 5.564.QQS issued to Weber et at di scloses a general system providing 
means for the user to first create a "time zone" correlated to a recording of an event, and then 
to flexibly enter into the time zone information relevant to the event addressing some of the 
disadvantages of the systems described above. 

These real-time data correlation and acces s systems have similar disadvantages. Tn the 
case of U.S. Pat. No. 5.564.005. the system does n ot automatically generate meaningful 
indica at the time the "time zones" are established bv the actions of the user. Rather. 
information about the events correlated to a time zone is entered later, in the form of 
handwritten strokes, or mav be generated and entered in real-time, subject to the conceptual 
and processing speed limitations and variance of a user's abilities. Therefore, the primary 
initial index guide to the meaning of the events within a time zone are the temporal 
characteristics of the ti me zone, augmented at the time bv the memory of the user. In all 
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these systems, no method or use r interface desi gn is explicitly suggested for handling the case 
of repetition or extension of certain events or activities in the recording, using a common 
indica for events or act ivities thereby related b ut occurring at different times, thereby 
facilitating or enhancing a user's personal note- taking stvle or accommodating the note- 
taking function to a variety of applications. For all except U.S. Pat. No. 5 .564.005. the user's 
ability to index notes to the recordin g is entirely controlled through the indicia, or notes, the 
user has entered in a document, since the time stamps or position captures are those made at 
the time th e notes ar e entered. Those systems assume, therefore, that the time of entry of a 
pote sufficiently corresponds with the time or positions of the recording to provide an 
adequate in dex into th e recording, requiring the user to mentally process the ongoing real- 
time events and within a certain ti me variance generate and enter the note. In the case of U.S. 
Pat. No. 4.924.387. this provides an adeq uate indexing structure since the stroke record made 
is intended to be a verbatim transcription of the verbal testimony made in a courtroom. In the 
case of EP 0495 612. however, where a verbatim transcription of the event mav not be the 
note-taker* s intention, such an assumption does not allow for the later, real-time 
augmentation of previously entered notes, while still recording, with additional notes related 
to the previously entered notes, since the later notes will be time stam ped with the time thev 
were entere d rather than with the time of the material to wh ich thev relate or are relevant. In 
the case of the system disclosed bv Rindfuss. notes mav be entered later, but the later-entered 
notes will be correlated with the po sition of the tape at the time the later thev relate or are 
relevant. In the case of the system disclosed bv Miller, notes addressed to the diskette record 
of the event at a later t ime are entered entirel y during a post-processing phase and not during 
the real-time recording of the event In some of these systems, neither time stamps nor tape 
positions are visible to the user and so the temporal or spatial context of the entered indicia is 
not available to the user to enhance the retrieval function. In addition, in EP 0 495 612 the 
function provided for creating topic or kev words from the entered indicia mav be practically 
limited to one display "page" or screen unless th e user reenters the topic o r kev word on a 
second screen or scrolls between screens to add a mark to a previously en tered tonic or kev 
word. As with augmentation of notes in general, there is no facility for associating a kev word 
or topic name created at a later time with notes entered earlier. 

Existing post-processing (non-real-time^ annotation systems in the field of post- 
production video editing provide for the creation of annotations about scenes correlated with 
"in" and "out" time codes identifying the scenes on a video recording. U.S. Pat. No. 
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5.21 8.672 is an example of such a system. Tt is disclosed th ere that scene descriptio ns may he 
revised after initial creation, but the correlation of the annotations are c onfined temp orally to 
the identified scenes. 

In the post-processing system fo r the correlation of legal depositions with video 
recordings th ereof disclosed in U.S. Pat. No. 5. 1 72.28 1 . a time code number is assigned bv an 
operator of the system to both the computer transcript and the videotape segment where each 
Question and answer passage begins, requiring manual time code entry. The location of 
individual w ords in the transcript mav also b e correlated with their corresponding position in 
the video re cording, but there is no such result during the real-time process. In addition, the 
system does not appear to provide for the ent ry of notes or annotations. 

Thus, despit eDe£Bjt£ the number of such prior art methods, a-needjhsrg exists a need 
for a na recorded information system that will digitally record events , process e s, or other 
observabl e ph e nom e na, and simultan e ously, e fficiently, and accurat e l y and automatically 
generate informative indexe s indica that are correlated in real-time? with minimal interference 
te-the recording of the events without affecting an interviewer's or observer's ability to take 
notes or assess the flow of events. There is a more - general need for systematic, structur e d, or 
standardiz e d obs e rvation and ass e ssment measur e m e nt methods, and ther e- is a need to 
generate standardized, systematic, and quantitative ind e x e s that ar e l e ss subj e ct to individual 
variono e and which can bo used between and among r e cord e d information p e rtaining to 
different individuals or ev e nts so that comparisons and other analy se s can b e mad e . 
Furthermore, it Tt would be desirable to have a system that allows subs e qu e nt annotation of for 
annotating an indexed digital recording of information in such a manner that the annotation 
did not disrupt the standardization of the index or otherwise adversely affect the ability of one 
set of recorded information in a standardized index key to be compared to another set of 
recorded information that - uses h aving the same standardized index key. 

There is also a need to generate a standard set of indexes that are not subject to 
individual variance and which can be used between and among recorded information 
pertaining to different individuals or events so that comparisons can be made. In the case of 
video and audio logging tools, exclusively post-processing systems are inadequate for 
generating notes and relationships between recorded segments, and are time consuming 
because thev require review of the entire tape in order to generate an index. Automatic 
indexing of video and audio notes bv system t ime stampi ng of key- or handwritten strokes or 
automatic detection of speaker voice changes does not provide adequate context markers for 

12 
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the recorded sign als or does not permit user control o f the amou nt of detail t o be captured. 
For example, an index created on die basis of speaker segmentation of the material would tell 
who was sneaking but not the substance of the talk. Establishing time zones through the 
action of a user does not directly generate indica containing information regarding the events 



SUMMARY OF THE INVENTION 

OBJECTS AND ADVANTA GES 
Accordingly, it 

It is an object and advantag e of the pr e s e nt inv e ntion to combine syst e matic, 
standardiz e d observation and ass e ssm e nt using pr e d e fin e d d e scr i ptors or m e asur e m e nt it e ms 
with simultan e ous recording to automatically g e nerat e m e aningful ind e x e s for th e recording 
in - real - time. Additional objects and advantages are: 

— to provid e a system and method that supports th e generation and storage of new 
d e scriptor or m e asurem e nt it e ms with rating scal e s for subsequent us e in syst e matic 
ob s ervation and asse ssme n t;- 

— to provide a system and m e thod that can import and stor e a s e t of pr e- d e fin e d 
d e scriptor or measurement items, such as an existing questionnaire or s et of rating scales, for 
subsequ e nt us e - in - syst e matic observation and ass e ssm e nt; 

to provid e a syst e m and m e thod that us e s pr e e xisting d e scriptor or m e asurement 

it e ms to structur e th e obs e rvation and ass e ssm e nt of e vents; 

— to - pro ¥ ido a system and method that uses measurement hems that employ 
quantitativ e rating - scales; 

— to provide a syst e m - and method that efficiently oreat e s ^ ui - index for any typ e of 
recording by fully automatic correlation with the us e of pre e xisting descriptor or 
m e asurem e nt it e ms; 

— to provid e- a - system and method that stores recorded material in digital form in 
c e ntraliz e d electronic storag e ; 

— to provid e a syst e m and m e thod that stores descriptions or ratings of observ e d and 
ass e ssed e vents or phenomena in centraliz e d electronic storag e ; 



particular pr e e xisting m e asurement item — in other - words, an observer can repeatedly a s sess 
or rat e a ph e nom e non or a r e curring event using th e sam e m e asur e m e nt it e m; 
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— to provide a s ystem and m e thod that allows unlimited concatenation of ind e x e d 
r e cord e d mat e rial when one or more phenom e na or r e curring e vents ar e r e p e atedly assessed 
using th e sam e m e asur e ment item; 

— to provid e a system and method that can - us e any part or parts of th e content of 

pr ee xisting descriptor or measurement it e ms 09 ind e x labels or k e y words; to provid e a 

system and m e thod in which a standard ind e x, bas e d - upon - a - specific questionnair e or 
measur e m e nt instrum e nt (i.e., a set of measur e m e nt - items), can b e appli e d across multipl e 
r e cordings; 

— to provid e a system and method that oon search, sort, and/or retri e v e r e corded 
materials based upon meaningful words or combinations or words from th e full t e xt of th e 
descriptor or mea s urement items used in th e ind e x; 

— to provide a system and method that can s e arch, sort, and/or r e tri e ve r e cord e d 
mat e rials bas e d upon numerical ratings from s tandardiz e d obs e rvation and quantitative 
assessment using m e asur e m e nt it e ms with rating scales; 

— to provid e a syst e m and m e thod that p e rmits ass e ssors to ro order recording s of 
e vents and compar e similar e vents from diff e r e nt subjects on th e basis of th e assooiatod 
m e asurem e nt items in ord e r to inv e stigat e possibl e r e lationships and/or tr e nds not previously 
disc e rnibl e ; 

— to provid e a system and method that allows a user^ such as a stud e nt or train ee , to 
vi e w and/or r e vi e w prior assessm e nt results, i. e ., d e scriptor or measurem e nt item a s s e ssm e nt 
choic e s, by - tho same user or by a diff e r e nt us e r, in parall e l with playback of the recording of 
the observed and assoas e devents or phenom e na, allowing for e ffici e nt us e of actual 
ass e ssment for teaching or training; 

— to provid e a system and method that allows playback of r e cord e d mat e rials with 
th e prior s e l e ctions of d e scriptors - or ratings hidd e n from display; 

— to provid e a syst e m and method that allows r e- as se ssment or r e rating during 
playback - fio that rater e ffici e ncy, accuracy, and r e liability can - be measured; 

— to provid e a syst e m and method that allows multiple users to make ratings bas e d 
upon a single original r e cording, and that allows multiple users to playback and rat e at th e 

Jul I1U uhiv, 

— to provid e a syst e m and m e thod that supports comparison - of - ass e ssm e nt results or 
ratings across multiple raters, subj e cts, events, or geographical sit e s, for analysis, teaching, 
monitoring assessm e nt, or other res e arch purposes; 
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— to provide a system and m e thod that can automatically compar e ratings to make - it 
e asi e r to t e ach, monitor, and conduct research; 

— to provid e a syst e m and - method that allow s unlimit e d manual addition of ind e x 
points and lab e ls during playback: and invention to provide a system a n d method fo r digitally 
recording information about an event or indi vidual and ge nerating a standard set of indexes 
for this information. 

It is a further object of the invention to provide a system and method for annotating 
an indexed , digital recording of information without changing the sta ndardized index 
markings in such a manner that would adversely affect the ability of one set of recorded 
information in a standa rdized index key to be compared to another set of recorded 
information having th e same standardized index kev. 

In accordance with a preferred embodiment of the present invention, the invention 
employs a system that includes: (a> means for presenting a subject or operator with a 
measurement item: (b\ a recorder that records and stores observations of events associated 
with the measurement item in a digital recording that includes a data track sequenced bv a 
time track of time data: and (c) indexing software that enters an index mark based on said 
time data into an editable database denoting said measurement item. Anv annotation, cross 
referencing, or entry of notes will occur in the database in association with the time data track 
index without affecting the index. 

In use, the testing system of the invention includes the following steps: (aft presenting 
a measurement item to a subject o r operator: (b"l digitally recording on a digital storage 
medium at l east one event associated with said measurement item, said digital recording 
having a event data track of event data sequenced bv a time track of time data: (c) indexing 
said digital recording bv entering time data related to said event into a field on computer 
database along with an identifier of said measurement item: and (d\ optionally, augmenting 
said computer database bv adding information to said database that relates to said event field 
without changing the relationship of said time data to said event data. — to provid e a system 
and m e thod - that - all o ws ass e ssors and others using a 't e st and respons e " method of 
inv e stigation great e r not e- talcing and annotation e fficiency ond flexibility than waspreviously 
available. 

The system and method of the invention permit measurement items to be associated 
to observed events, the responses recorded and indexed bv measurement items onto a 
computer-based database for review, annotation and augmentation of diverse information 
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without affecting the time sequencing of the recorded data or the ability to replay a portion or 
all of the re corded eve nts. Such a system will allow assessors and others using a "test and 
response" m ethod of i nvestigation much greater note-taking and annotation flexibility than 
was previously available with prio r systems. The system of the inventio n also nerniils 
assessors to re-order r ecordings of events and compare similar event s from different 
subjects- on the basis of the associ ated measure ment items in order to investigate possible 
relationships and/or trends not previously flswmfck, 

Furth e r obj e cts and advantages of th e inv e ntion will b e com e apparent from a 

consideration of th e drawings and e nsuing discussion. 

SUMMARY OF THE INVENTION 

The present invention combin e s syst e matic obs e rvation - and assessment using 

predefined descriptor or measurement questions or it e ms with - simultan e ous digital recording 
in ord e r to automatically g e nerate meaningful ind e x e s that support improvements in 
e ffici e ncy, consist e ncy, and accuracy in r e tri e ving and utilizing the recorded mat e rial. Based 
upon th e cont e nts of th e measurem e nt questions or it e ms, including any qualitative or 
quantitativ e d e scriptions or numerical rating r e sults from us e of th e d e scriptors or 
m e asur e ment items, the indexes con readily bo furth e r - proces s od - and - manipulated to 
additional advantag e . Th e inv e ntion providos - a - mothod for access and manipulation of digital 
recordings of e v e nts by uoo of thoso pro entered or pr e e xisting measurement items as 
m e aningful ind e x information d e signators such as k e y words, lab e ls, item numbers, or 
num e rical ratings. Such a m e thod is particularly used for process e s r e lating to syst e matic 
interview or assessment method s , including th e training and monitoring of int e rvi e wers or 
ass e ssors and the storage, - retrieval, analysi s , and oth e r manipulation of r e cordings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

In th e drawings, closely - related figures have the same number but diff e r e nt alphab e tic 
suffix e s. 

Figs. 1 A and IB show an example of a measurement item from the World H e alth 
Organization Schedules for Clinical Assessment in Neuropsychiatry. 

FIG. 1 is a flow diagram of the recordi ng and indexing functions of the present 
invention. 

Fi eFIG . 2 is a gen e raliz e d bloclc flow diagram illustrating th e configuration of the 
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syst e matic assessm e nt and simultan e ous recording and automatic - ind e xing functions of a 

pr e f e rred e mbodim e nt - of th e pr e sent invention. 

Figs, 3A and 3B - illustrate furth e r d e tails of th e syst e matic assessm e nt and 
simultan e ous r e cording and automatic ind e xing functions of a pr e f e rr e d e mbodiment 

of thepr e sent inv e ntion.Fig. A i s a block diagram illustrating th e configuration of th e search, 

sort r e tri e vn L of the playback? and manipulation editing function s of a pref e rred e mbodim e nt 

of th e pres e nt invention. 

Fig. 5 is a block diagram illustrating th e configuration of - post processing syst e matic 

r e ass e ssment functions of a pr e f e rred embodim e nt of th e present inv e ntion^ Figs> 6 A and 6B 

show scr ee n shots of a pr e f e rred embodim e nt of the present invention. 

DETAILED DESCRIPTION 

The present inv e n t i o n is a- m e ans and m e thod for automatically g e n e rating m e aningful 
indexes for recorded material. In the following detail e d d e scription, num e rous sp e cific details 
ar e s e t forth in ord e r to provid e a thorough understanding of the present invention. Howev e r, 
it will b e appar e nt to on e of ordinary skill in th e art that th e se spec ific detail s need not be 
us e d to practic e th e pres e nt inv e ntion. In other instances, well known structures, int e rfaces, 
and processes hav e not b ee n shown in d e tail in ord e r not to unn e c e ssarily obscure th e pr e s e nt 
inv e ntion. 

A preferr e d embodim e nt of the invention will now b e d e scrib e d h e rein b e low with 
r e f e r e nc e to th e drawings. To e xplain cl e arly, explanation will b e made in accordanc e with 
the following ord e r: 

(1) with r e spect to conducting an ass e ssm e nt int e rview; 

(2) with respect to playing back an ind e x e d int e rvi e w; 

(3) with r e spect to s earching and r e tri e ving ind e x e d mat e rials; 
(A) with resp e ct to re-assess ing-a prior assessm e nt int e rvi e w. 

(1) Conducting an as s es s ment intervi e w 

Figures 1 A (item) and4B-(rating scale) illustrat e ' a typical measur e m e nt item from 
the World Health Organization (WHO) Schedul e s for Clinica l Ass e ssm e nt in 
N e uropsychiatry (SCAN). In general, measur e m e nt it e ms typically ar e compos e d of: 1) 
definition of a phenomenon of inter e st; 2) instructions regarding observation of the 
phenomenon; and 3) a quantitative rating scal e . Suitabl e measur e m e nt it e ms can vary ov e r a 
wid e rang e and will depend on the information sought from th e t e st subj e ct and th e typ e of 
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analysis int e nd e d. Exampl e s of analysis includ e b e havioral r e spons e s to audio and/or visual' 
information (qu e stions, directions, light s ^ s ound s , etc), physical e xaminations of laboratory 
specim e ns, oral d e positions of witnesses, etc. Th e SCAN r e presents a detail e d instrum e nt for 
clinically - bas e d, standardized psychiatric assessm e nt of psychopathology - ph e nom e na. - Th e . 
SCAN contains over 1,500 measur e m e nt it e ms, e ach of which is composed of at l e ast an item 
number 102, an item titlo 10 4 , and an item rating scal e 120 (Fig. IB) with^ating scale toxt 
122 and rating scale quantitative ratings 12 4 , and optionally - on item d e finition 106ran-item 
glossary e ntry 10 8 providing furth e r d e finition, sugg e st e d prob e s 1 10 (questions) for 
int e rview us e , and oth e r instructions 1 12, all of which combin e d ar e referred to as the 
"cont e nt" of a SCAN m e asur e ment it e m. Each m e asurem e nt it e m as a whol e with its 
constitu e nt - cont e nt may b e consid e r e d as an obj e ct with r e sp e ct to its function for conducting 
an assessment int e rvi e w and ind ex ing -a-s imultan e ous r e cording. 

Figure 2 shows a gen e raliz e d - block - diagram of a pr e f e rr e d e mbodim e nt of th e pr e s e nt 
inv e ntion. In this typical archit e ctur e , th e syst e m display pr e s e nts a sot of predefined 
m e asur e m e nt items 202 to th e user, while simultaneously proc e ssing and digitally r e cording 
th e obs e rvabl e e v e nt to b e rat e d, in this cos e th e psychiatric int e rvi e w 204. As the us e r carries 
out syst e matic obs e rvation and ass e ssm e nt with s e l e ction of a m e asur e m e nt item 206, 
optionally g e n e rating m e asur e m e nt result data by uoing th e rating - scal e of th e measurement 
it e m, th e syst e m automatically d e finos - a - aogmont of recorded matorial-20 8 ^ bas e d - upon th e 
repoatod 214 s e lection by th e user of m e asurem e nt items from th e display e d set ov e r th e 
cours e of obs e rvation and assessm e nt Th e e nd of th e pr e vious s e gm e nt and th e start of th e 
next s e gment is d e fin e d wh e n a m e asur e ment it e m is s e lected. For th e first s e gm e nt, no prior 
segment end is defined, and th e start of th e s e gm e nt is d e fined by th e start of th e digital 
recording r- Each segment i s indexed to th e currently select e d measur e m e nt - it e m - that d e fin es 
the s tart of th e s egment 210. The combin e d segm e nt d e finition and segment indexing 
functions 212 are carried out in th e background without us e r observation. Th e digital 
recording, m e asur e m e nt result data, and ind e x information or e stor e d in a data base 220. The 
index information thus includes the t e xt cont e nt 102, 10 4 , 106, 10 8 , 110, 1 12, and 120, of 
e ach associat e d m e asurement item and the ratin g-sea le4 - 2frresults 122 and 124» which can be 
us e d as a common set of k e y words, labels, - and - quantitative attributes to identify index 
e ntries and allow r e tri e val of diverse recorded event data from different subjects under a 
common fram e work. Th e r e sulting assessment - form s- a data base file that can b e stored, 
shar e d, or int e grat e d into other digital records as need e d by the us e r. 
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Figur e s 3 A and 3B illustrat e further detail s of th e syst e matic assessment and 
simultan e ous recording and automatic ind e xing functions of a pr e f e rred embodiment of tho 
pres e nt invention. Th e syst e m provid e s th e user with a set of pr e d e fined measurement items 
and m e ans for navigation and s e l e ction 302. Simultaneously, a process of digital recording 
occurs 304. When a measurement - item is s e l e ct e d 306, th e syst e m automatically gets lower 
l e v e l addr e ss information such as a tim e cod e to id e nt i fy th e start of th e r e cording segment 
30 8 , and links th e segmont start address with th e current m e asurem e nt it e m 310. 

Whil e th e s e addr e ssing and linking process e s occur in th e background, th e user 
assess e s th e subj e ct, optionally choo s ing to us e th e rating scal e from th e current measurement 
it e m to g e n e rat e m e asur e m e nt rating data 3 12. Wh e n th e n e xt measurem e nt it e m is s e lected 
by th e us e r 314, th e syst e m automatically gets lower - lev e l address information to identify th e 
end of th e r e cording s e gm e nt 3 16, cr e at e s an index item by linking recording segment sta rt 
and e nd identifi e rs to th e e mployed m e asur e m e nt it e m, along with any measurement results 
data 320, and then stores tho index - item in the data bas e 220. If th e m e asurem e nt item was 
pr e viously us e d, and th e r e for th e r e is an existing ind e x it e m 324 using that m e asur e m e nt 
it e m, th e n th e r e cording segment start and e nd identifiers ar e app e nd e d 32 8 to th e existing 
ind e x it e m and stor e d in th e data baso 220. 

Figure 4 is a block diagram illustrating th e configuration of th e query, retri e val, 
manipulation, playback, and index i ng functions of a pr e f e rr e d embodim e nt of th e pr e s e nt 
inv e ntion. On e means of r e tri e ving recorded material is through us e of th e pr e d e fin e d 
m e asur e ment items 402 that were us e d in th e creation of th e- ind e x it e ms. From th e display, 
the*%is©r can vi e w and s e l e ct a measurem e nt item. Likewis e , th e data bas e can b e qu e ri e d by 
s earching th e ind e x it e ms for k e y words from th e content of th e associat e d m e asurem e nt 
item s 4 0 4 . Th e data bas e can - also b e queri e d by s e arching for specified assessment results 
4 06. From th e s e action s ^ - e ith e r singly or combined 408, tho s y s t e m automatically g e n e rat e s a 
qu e ry 410 to th e data ba s e 220 that r e tri e v e s 4 12 th e associated segment or segm e nt s of 
r e cord e d mat e rial for further management 416. Any search results can b e sorted or filtered 
using standard means, and can be generalized to s e arch across multipl e subj e cts and multiple 
assessments usin g- standard means. Retriev e d r e cord e d mat e rial can be s elected and 
manipulated for output to another system or fil e , or can b e played back with standard 
functions 418. During playback, additional ind e xing can b e p e rformed. If furth e r ind e xing - is 
d e sired by the user 420, the syst e m supports a choic e 422 of cr e ating and using new index 
lab e ls 424 or using existing measur e m e nt items 426. Th e ind e xing proc e ss 42 8 d e scribed 
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earlier associates th e s e ind e x e lem e nts with r e cord e d mat e rial^ and th e r e sults ar e stor e d 130 
in the data base 220. 

Figure 5 is illustrat e s th e configuration of post processing systematic r e assessment 
functions of a pr e f e rr e d e mbodiment of th e pres e nt invention. A r e assessment mode 502^ ro 
ass e ssm e nt subj e ct 501, and an existing assessm e nt 506 can b e s e l e ct e d 502, allowing the 
obs e rvation through playback of th e r e cord e d materials association with th e e xisting 
assessm e nt 506. Th e set of pr e d e fin e d m e asur e ment that was used to carry out th e original 
ass ess ment i s display e d 50 8 , allowing its us e for navigation and s e l e ction of specific ind e xed 
se gm e nts of r e cord e d mat e rials. Standard playback functions or e also availabl e 512. The 
mat e rial s ob s erv e d through navigation and - playback can th e n b e r e ass e ss e d using th e s e t of 
pro defined measur e m e nt herns 51 4 , and the re - assessment result s stored 516 in the data bas e 
220. Standard manag e ment functions can th e n be used to process - the - stored data 520i 

Figur e s 6A (ass e ssm e nt) and 6B (playback) illu s trate apreferred - embodiment of the 
present invention as impl e m e nted in a working softwar e program (^liniM e trica"). For 
ass e ssment, a set of pr e- d e fin e d measur e m e nt it e ms is display e d 602, from which th e user can 
sel e ct flic curr e nt - m e a s ur e m e nt it e m 6 Mi The rating scal e t e xt 60 8 and quantitativ e valu e s 
606 allow systematic rating. Digital - audio and video recording can b e s tarted 620, and will b e 
automatically ind e x e d in s e quenc e with th e s e l e ct e d m e asurem e nt items. For playback, th e 
list of m e asur e m e nt it e ms us e d in th e ass e ssm e nt is display e d 632, allowing retrieval of the 
corr e lat e d se gm e nt - of r e cord e d - mat e rial, for playback via a standard s e t of playback controls 
640. Furth e r, th e ind e x e d record e d mat e rial can b e r e tri e v e d through t e xt search of all of th e 
m e asurement it e m, including rating s cal e , text 638. Th e r e cord e d mat e rial that is play e d back 
can bo ro asse s sed 650.Th e invention is described in d e tail invention is described with 
reference to inquiries from an assessor and responses made by an interviewee during an oral 
intervie w assessment . It will be understood that the description is equally applicable to any 
form of systematic assessment or observation by verbal, visual, audio, or physical method 
(e.g., questions or directions whether presented orally or in tex torinted form , mental or 
physical tests or testing, us e of s oftwar e or hardwar e , etc.) and any -form of response by the 
subject, human or other, of observation (verbal, nonverbal, or physical response) with mere 
changes in semantics to describe the particular form of test and response. 

A gen e ral testing system according to the invention includes the following steps: (a) 
presenting a s e t of pr e d e fin e d m easurement itea sitem to a subject or operato r for use i n 
sfr ucturing and carrying out th e observation and ass e ssment of ev e nts or ph e nom e na of 
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inter e st; (b) simultaneously making a digital r e cording of one or mor e asp e ct s of th e obs e rved 
e v e nts or ph e nom e n a : (b) making a digital record on a digital storage medium ? at least one 
event associated with said measureme nt item, said d igital r e cording inoludin g record having 
an event track of event data sequenced by a time track of time data; (c) automatically 
indexing said digital recording b y automatically entering time data aftd-an identifier relating 
to the curr e nt s e l e ct e d m easurement item into a r e oord in fi elds on a computer data 
bas edatabase ; and (d) optionally, augmenting said computer data bas edatabase bv adding 
informatio aformation to said data bas ed atabase that relates to said measurem e nt items or said 
digital r e cordin g event field without changing previously establish e d ind e xing r e lationships 
and - data bas e r e cordsthe relationship of said time data to said event data . 

An interviewing a s sessment system according to the invention includes a personal 
computer having: (a) an output communication subsystem that allows communication with an 
intervi e w e r or interviewe e or interviewer so that pr e d e fin e d measurement items can be 
presente d and s e lect e d^ (b) an input subsystem for the interviewer or interviewee to respond 
to the measurement item communication s to generate assessment data result s; (c) a recording 
subsystem tha t simultan e ously makes a digital record of the audio, video, physiologic, or 
other types of r e cordabl e e v e nt data from e ith e r or both of tho int e rvi e w e r's and outout 
communications as well as the interviewee's b e haviors and r esponses? with a data track of 
said-event data synchronized tohy a time track of time data; (d) an ind e xing subsystem that 
automatically gen e rat e s corr e lation entries that defin e time segment s of th e digital r e cording 
bas e d upon - th e-se lection or pr e s e ntation of succ e ssiv e measurement it e m s , and links thos e 
tim e s e gments to tho moa s uroment items, th e r e by creating a m e aningful index - for th e se 
s e gm e nts of digital recording by us e of th e contents of th e corr e lat e d measurement items and 
any quantitativ e m e asur e m e nt rating results; and ( e >a digital storag e r e tri e val subsystem that 
us e s th e m e asur e ment it e m bas e d index to retri e v e segm e nts of r e cord e d mat e rial for 
playback, analysis, e diting, annotating, ro rating, or other r e lat e d puroos e s a digital storage 
subsystem that allows the digital record to be stored for playback and editing: and (e\ a 
com puter with database software that can generate a database event index record based on 
said time data for each measurement item . 

The means for presenting the subject or operator with measurement items can take 
many forms? depending on the type of measurement item involved with the assessment. The 
system and its method of use is most easj |v described^ere in the context of an interview 
between a subject and an interviewer using a computer to display a series of measurem e nt 
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item-questions to the interviewer as a guide for the interview. At th e sam e tim e , thelfag 
interview is recorded by a audio or vid e o recorder (preferably digital) that includes a data 
track synchronized by a time track of time data. As the interviewer moves from one question 
(i. e .- m e asurement item) to a n e w - oue s tio n to another via input ifitejQ the computer, this 
qu e stion switohingt hs input is an act that causes the data base - (either from softwar e-er 
hardwar e functionsl database to record the time data (preferably a stop data bit for termination 
of the preceding time-segment of recording and a start data bit for beginning the next &ne 
segmen t of r e cording) and an associated identifier that links th e tim e segment of 
r e cordin g pertains to the new question (measurement item) , th e r e by g e nerating an index e ntry . 

As those skilled in the art will appreciate, the indexing act can be done in an active 
interview situation, by the interviewer, as questions are posed The indexing can also be done 
in passive situation, where the operator is observing but not necessarily actively interacting, 
by the operator entering meut san input into the computer from a list of predefined topic items 
(measurement items) during actions by the subject of observation. Either method will 
generate a data bas edatabase of event data that is indexed by a series of measurement item 
based-identifiers that can be sorted and retrieved for selective retrieval and playback of the 
events. 

Th e proforro dP referablv. the software e mbodimcnt - of th e inv e ntion for a personal 
computer will havo that has the capability to: (i) communicat e pr e defined inquiries to the 
interviewee directly or through an interviewer in a systematic manner^ (ii) provide ajating 
soalos or methods associat e d with or part of th e or e d e fin e d inauiri e s scale or method bv 
which said inquiries can be rated or applied for th e purpos e of describing said interview ee ^ 
and (ii ft automatically oonorato provide an index in the form of retrievable^ index marks 
associated with the pr e- d e fin e d inquiries. A personal computer is used to communicate the 
inquiries and at th e same tim e fl tgpjg capture and digitally record or control a recording 
device that records the interviewee's responses to form a digital record of indexed inquiries 
and indexed responses. As thos e skill e d in th e art will appreciat e , any type - of - computing 
d e vic e could b e us e d to provid e th e s e functions, including customized or pr o grammable 
hardwar e that provid e s functions describ e d obov e a s provided - by software. 

If digitally r e cord e d D uring later playback and if responses are to be further indexed 
during lat e r playback, the system can also insert additional response index marks and the 
identifiers for th e ind e x marks them into the digital record so that digital materials ar e furth e r 
ind e xed for r e tri e val. For e xampl e , in r e vi e wing an oarlio n ssponses are digitally marked for 
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retrieval. An assessor mav also insert additional inquiry index marks into the recorded 
interview;- an a s ses s or may wish to employ pr e d e fined measur e m e nt it e ms not us e d 
originally in interview, to furth e r ass e ss or rat e th e int e rvi e w. In addition to use of pr e d e fin e d 
m e asurem e nt items which provide m e aningful t e xt content thats e rv e s - as ind e x lab e ls or k e y 
words, new indox^ Jndgx marks can be placed at any point in the recording with-manual entry 
of a lab e l or key words. Furth e rmore, if the user wish e s for examplo - to - study or analyze 
communication and response patt e rns, n e w m e asur e ment items - could - be created and stored 
for r e p e at e d us e . Pr e f e rably, an "inquiry" m e asur e ment item oould be created, stor e d , and 
th e n ins e rted at each location of interestbut m mmiY intox mark i? PP?feraWY inserted 
where a specified data item inquiry is being communicated, and a Response " m e asurem e nt 
item could be created, stor e d, and index mark is inserted where the response is made. The 
inquiry and response index marks are desirably standardized and relate to the syst e matic 
content of the inquiry or the response. 

The invention is conveniently explained in the context of an interview under the 
WHO Schedules for Clinical Assessment in Neuropsychiatry (SCAN). These schedules 
represent a detailed instrument for clinically-based, standardized psychiatric assessment of 
psychopatholog y phenomena. Th e SCAN contains ov e r 1,500 measurement it e ms, each of 
which is composed of at least an it e m numbon an item title, and an it e m rating scale, and 
optionally an item definition, an item glossary entry providing - further d e finition, sugg e st e d 
prob e s (questions) for intervi e w us e , and oth e r instructions, all of which combined ore 
r e ferred to as th e "cont e nt" of a SCAN m e asur e m e nt item. An e xampl e m e asur e m e nt item 
from - the SCAN is shown in Figs. 1A and IB . The SCAN is a widely recognized and used 
international standard , having b ee n translat e d into ov e r 25 languag e s . It will be understood by 
those skilled in the art that the techniques applied to a SCAN assessment would be applicable 
to other types of assessment instruments, physical examinations, or other physical 
phenomenon whose analysis is aided with a recording of sights, sounds, phy s iology, and/or 
other domains of observation with only minor semantic differences that describe the same 
essential types of events. 

U s ing th e present invention to oonduct a SCAN psychiatri cD uring an assessment 
interview with a human subject, real-time selection bygf the interviewer of a relevant SCAN 
item in the natural course of the interview, observation, and assessment of clinical 
phenom e n a events automatically generates a series of t ime-base dstamned inquiry index entFy 
fedanarks that allows us e of the SCAN K e m serve as an informative indicum for the 



23 



Received from < 703 903 9000 > at 10/6/03 3:18:53 PM [Eastern Daylight Time] 



10- 6-03; 3:53PM;MILES STOCKBFt I DGE ; 703 903 9000 # 25/ 54 

Docket No. G2622-908496 i 
Appl.No. 09/539,772 

Preliminary Amendment filed with RCE Request 

simultan e ously re corded dat a (e.g., any of audio, video, physiologic) . The time-stamp index 
marks (time stamp plus identifier) are stored in data bas e database fields with time data that is 
correlated to a sequence of event data in the recording of the events. The resultant set of time 
stamp index marks then provides a general index structure which can be used to further 
support, independently of the real-time process, a flexible note-taking structure and user 
interface that complements diverse personal note-taking styles and application needs. 
* — ^> Notations and annotations in the form of text, sound, video, or other data can be 
entered into the data bos e database as separate fields of information without affecting the 
relationship between the time stamp index mark and the recorded data associated therewith. 
Thus, the use of digital index marks of an event recording in an editable data bos edatabase 
provides assessors with the ability to generate assessment records rich in diverse content. 
When coupled with standardized assessment inquiries (e.g., a list of standard interview 
questions) and a data bas e database field that identifies each inquiry by a standardized 
descriptor for each inquiry, this rich a ssessment record can be used as a basis for 
comprehensive comparisons on a rich, d iverse content. 

In the case of assessment interviews in which there is no pre-existing set of 
standardiz e d m easurement items (questions, definitions, inquiries, instructions, directions, or 
other tasks), an important feature of the present invention is the ability for the assessor or 
another individual (e.g., a n interview designer) to generate a set of items prior to the event to 
be obs e rved, assessed, and r ecorded . This supports th e proc e ss of dovolopin e and develop a 
new set o f standardiz e d inquiries for investigating new issues. This set of items can be used 
and re— rused as real-time index information designators which can later be used as standard 
access mechanisms into both the notes and the correlated recorded signals. These new 
inquiries, e.g., key words, labels, or any other pointer to information, can identify specific 
concepts in the user-produced information tefor which the user wants to have later access. An 
information designator is treated as an object that ean-bejs easily created and reusedr-Tht s and 
this novel implementation both in the user interface and data structure e&r the present 
invention provides a flexible and powerful indexing tool fe finlQ the recorded signals in 
addition to the access provided by the recorded medium's address parom e ter s parameter . 

In an environment where a recording of events is being made simultaneously-wkh-a 
th e rating of those events, the interviewer has limitation s, du e to limitations of human 
cognitiv e abilities, - on what actions can be taken or what-additional items ean-be-considered as 
the interview is in progress. The present invention produces a digital data base d atabase of 
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indexed inquiries and responses that can be used to organize the assessor's notes T originally 
written in free text or graphical form? but later entered into the data bas edatabase with 
minimal restriction as to the amount of that information or when it is entered. The inquiry 
and/or response index marks allow the assessor's notes to be correlated with the recorded 
signals of the event. Further, the present invention provides the assessor with the ability to 
modify any notes that were previously entered without regard to when the modifications are 
made or where they are located on the recording medium. 

The invention recognizes, moreover, that the temporal or spatial sequencing of 
information is itself often a significant clue as-to the content of the information and may aid a 
user in recalling memories or in retrieving recorded information from a perceived event The 
inquiry and response index marks of the present invention provide a mechanism for the user 
to control the playback sequence and thereby view the index with items sorted in, e.g., 
chronological order, type of inquiry (as denoted by the type of inquiry index encoded into the 
digital record), by type of response (also as denoted by the type of response index mark 
encoded into the digital record), as well as any other order desired by the assessor. 
— The system of the present invention is particularly effective for video and audio 
interviews where the events include a time sequence of responses by the interviewe e or t est 
subject. These events are recorded as a series of audio or video input data s e gm e nts 
sequenced by a time track of time data. Because the amount of detail that may be captured 
for a particular time on an associated recorded medium is open-ended, a richer set of 
annotations may be e ffici e ntly mad e by us e of pro defin e d measuromont-itemsmadg while the 
video is being digitally recorded in real- time than is typically allowed in previous analog 
systems. Thi s r e li e v e s th e not e tak e r (i. e . th e measur e ment ass e ssor) of many of the pressures 
typically associated with taking intervi e w not e s in real tim e . Th e time s e au e nc eThe time zone 
and reusable information designator object features provide for treating the event as a whole 
and permi tb v permitting later augmentation or modification of previously made note s/This 
relieves the note-taker of many of th e pressures typically associated with taking interview 
notes in real time . 

It is another significant advantage that the system of the present invention operates in 
both real-time and post-processing environments. This feature recognizes that 
notetakin e note-takin q as a human activity is typically an interactive process. The output data 
structure (e.g., the ind e xed v ideo mat e ria^l ogL produced by the system of the present 
invention during a real-time note-taking session while the event is being recorded may be 
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further modified and/or augmented by the same user or by different users during post- 
processing review sessions occurring in parallel or seria tseriatim . This facilitates and 
potentially reduces the time r e auir e d involved in the p ost-processing session ss sssion . 

A system for correlating event data being recorded in real time to event information 
entered into the system by a system user includes input circuitry connected to a user input 
device for producing signals indicating actions of a system user; output circuitry connected to 
a display having area for presenting images; and a processor connected for receiving the 
signals from the input circuitry, and connected for providing images to the output circuitry. 
The system also includes a digital recording device in communication with a digital storage 
medium for producing a plurality of signals indicating recorded event data about an event for 
storage on the storage medium. The recording device is currently producing the plurality of 
signals and indicating the recorded event data and is connected for simultaneously providing 
address data items to the processor of the syste m and to the recording device . The recording 
device stores the plurality of signals indicating the recorded event data on the storage medium 
so that a recording address value of each address data hem provided by the address source 
indicates a portion of the recorded event data. The syste ms vstems also includes memory for 
storing data? including instruction data that indicat e indicating instructions the processor 
executes. 

Stated another way, the system may be configured to operate in a post-processing 
environment so that an event that has already been recorded but not indexed is the event 
about which a log is constructed according to the present invention. The event that has 
already been recorded is digitally stored on a storage medium as a plurality of signals 
indicating recorded event data about the event. In this aspect of the invention, the system 
includes an output device for presenting the previously recorded signals thereon to the system 
user. This output device is in communication with the storage medium including the plurality 
of signals indicating recorded event data about the event. The output device is connected for 
providing a plurality of address data items to the processor of the system as user-requested 
address values; each address data item is a recording address value indicating a portion of the 
recorded event data* Preferably, the plurality of signals indicating the recorded event data 
includes the plurality of address data items, but the output device may itself generate them as 
the plurality of signals is being presented by the output device. 

The system of the present invention permits a system user to interactively produce 
stored information that is correlated with an "address". The "address" refers to a 

26 



Received from < 703 903 9000 > at 10/6/03 3:18:53 PM [Eastern Daylight Time] 



10- 6-03; 3:S3PM;MILES STOCKBR I OGE ;703 903 9000 # 28/ 54 

Docket No. G2622-908496 « 
Appl. No. 09/539,772 

Preliminary Amendment filed with RCE Request 

sequ e nc emeasurement dimension used to identify a portion of the stored information. When 
the stored information is being correlated with recorded signals, the address typically refers to 
the location of the recorded signals on the storage medium storing the recorded signals. 
However, the system of the present information is not limited to correlating information with 
recorded signals, and may be used in any situation where later access to the stored 
information would be facilitated by correlation with an address, and by having an access 
mechanism as provided by the "information designators"* Generally, when the system user is 
producing a data bas edatabase of stored information, the address record will be time data 
taken from the time track of the recording of the events. 

The computer-controlled system of the present invention uses common, readily 
available hardware components. For an assessment interview delivered via computer, the 
system includes a computer with an image display that will display questions or directions, an 
input that receives signals from a user interaction device, e.g., keyboard, digital tablet, 
microphone, camera? or other recording device that produces a data track synchronized by a 
time track. The display can present the m e asur e m e nt it e m questions directly to the subject or, 
alternatively, can display a series of questions to an interviewer/assessor who then presents 
the questions to the subject. Some act taken to move from one question to another generates 
the data bas e database time entry or start/stop time entries as well as the index identifier that 
pertains to the question. 

Analog recording devices can hav e th e ir r e cordingsh g digitize d, or th e obs e rvabl e 
events can b e or recorded directly as a digital recording that includes: (a) a data track of 
sound, image, or other event data that are synchronized by (b) a time track that includes time 
data correlated with event data. The use and existence of these two data tracks is typically 
transparent to the user of the equipment but can be read by a computer and used to generate 
data bas ed atabase entries for each recorded event based on the time track data associated 
therewith from the digital recording. 

The user interaction devices useful in the present invention may include any one of a 
variety of input devices controllable by a human user. For example, the user interaction 
device may include a pointing and gesturing device to indicate actions, gestures, and strokes 
such as those produced when making handwriting gestures. A stylus or other form of 
handheld pen are examples of a suitable user interaction device. Other pointing devices such 
as a mouse , trackball, e y e position, or brain activity e l e ctrical or magn e tic patt e rns may also 
b e suitabl e for indicating actions, gestures, and strok e s^ or a trackball mav also be suitable for 
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indicating actions, gestures, and strokes. 

The present invention is fu rther understood bv reference to FIG. 1 which shows a 
flow diagram of the recording and indexing functions of the present invention. A digital 
recording 1 of a series of events i s made direc tly bv a digital recorde r or made into a digital 
formation 2 from an analog recording. The indexing sy stem 3 is u sed to generate a database 4 
of index items 5 based on measurement items 6. 

Suitable measurement items can vary over a wide range and will depend on the 
information sought from the test subject and the type of analysis intended. Examples of 
analysis include behavorial responses to audio and/or visual information (questions. 
directions T lights, sounds. etc.\ physical exam inations of laboratory specimens, oral 
deposition s of witnesses, etc. In die case of a behavioral examination, measurement items 6 
can be generated in process 10 from a standard assessment instrument 7. open ended 
questions 8 and anv oth er inputs 9 de sired bv the assessor. Preferably, process 10 is a simple 
system that is readily repeated. Suitable examples include a question outline or list. 

Similarly. FIG. 2 shows a flow diagram of the playback and editin g functions of the 
present invention. Digital data 20 made of an event sequence track of event data is 
synchronized bv an associated time track of time data is played back 21 for audio, video. 
and/or physiological events. Preferably, new index items 22 are created during a post- 
processing playback and review p hase but may also be created separately, for example as a 
pre-existing detailed set of measurement sub-items, or as additional not es during the original 
assessment process. Each index items is based on a link 23 between the time data in the time 
track with corresponding event sequence data and entered as index data 24 in a database. This 
database can include associated records that identify the in dex entries bv a common set of 
labels, index attributes 25 or other identifier that allow diverse event data from different 
subjects under a common framewo rk. The resulting assessment forms a database file 26 that 
can be stored, shared, or integrated into other digital records as needed bv the assessor. 

Although the foregoing disclosure relates to preferred embodiments of the invention, 
it is understood that these details have been given for the purposes of clarification only. 
Various changes and modifications of the invention will be apparent, to one having ordinary 
skill in the art, without departing from the spirit and scope of the invention. 
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